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Introduction 


This document presents standards for the planning and conduct of 
statistical surveys. The recommendations embodied in these standards 
have been developed by the Bureau of the Budget in consultation with the 
statistical agencies of the Federal Govermment. They are intended as- 
guides to good statistical practice for use in surveys conducted directly 
by Federal agencies and in those conducted, under Federal sponsorship, 
by universities, market research firms, trade associations or other con- 
tyacting organizations, 


These standards apply primarily to statistical surveys. As used 
here, a "statistical survey" is one in which the primary purpose of the 
individual returns is to supply data which can be aggregated to provide 
statistical or numerical information needed on a particular class, seg- 
ment, activity or area, and in which no administrative action is taken 
on the basis of the individual return itself, Among the data collections 
for which these standards have only limited relevance are forms used in 
applying a Federal law or regulation to an individual or firm (such as 
tax returns or the financial and operating reports required by regula~ 
tory commissions) and applications or registrations (such as forms used 
in applying for materials, grants, loans, licenses or other privileges). 
Also, many of the points covered in these standards will not be relevant 
for administrative surveys designed to collect summary information needed 
by an agency in administering a specific program (such as summaries of 
procurement or production status, many types of requirements estimates, 
or fiscal accounting reports from the States on prant-in-aid programs), 


It is recognized that the standards cannot be applied uniformly. or 
precisely in all situations. Special considerations may be involved in 
surveys of an exploratory, experimental or methodological type; or in 
pilot or preliminary surveys where the primary purpose is to evaluate 
alternative approaches or techniques, Similarly, no single survey design 
can meet all requirements. <A particular design should be that most ap- 
propriate for a given set of purposes, in relation to considerations of 
time and cost, Such considerations will help to determine, for example, 
whether to conduct a survey by mail or by personal interview, whether 
to use complete enumeration, some systematic sample design or "cut-off" 
procedure, or some other suitable device for selection of the respondents. 


The standards are presented as guides for the planning and conduct 
of surveys, and not as a requirement of additional materials for Bureau 
of the Budget review of plans and report forms under the Federal Reports 
Act. In submitting materials to the Bureau of the Budget for review 
under the Federal Reports Act and Budget Circular No. A-liO, Revised, the 
agency need supply only the information and attachments already required 
by Standard Form 83, During the review process, however, the Bureau of 
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the Budget may in specific instances ask the agency to furnish addition— 
al information on its consideration of appropriate standards, 


Survey Standards 


1. Purpose of the survey 


Before any other steps are taken in planning a survey there should 
be a clear understanding of the precise purposes to be served in terns 
of information to be collected, hypotheses to be tested, or problems to 
be solved. It is helpful in reaching such an understanding to prepare 
& concise statement of the quantitative information needed and the exact 
purpose to. be served. 


22 Relation to other Surveys or programs 


Before the scope and content of a survey are determined, prior work 
and current activities in the field should be reviewed, and the problem 
defined in relation to similar programs. Earlier Surveys on similar 
topics should be analyzed, and relationships to existing data fully ex- 
plored. 


i: 3e Development of the survey plan 


Decisions should be reached on each of the following points in terms 
of the purposes to be served, the costs involved, the time required, the 
degree of precision needed, administrative Limitations (established prac 
tices, location of field offices, ete.), and need for including or ex- 
cluding specific areas for nontechnical reasons: ate 


& Respondents. The respondent group should be defined in terms of 
the classes, segments or areas to be included in the survey. 
In selecting the respondent group, attention should be given 
to the availability and nature of source materlalsy such as 
mailing lists, maps,. directories. 


b. Extent of coverage, Within the respondent group selected, it 
must be decided whether enumeration should be complete or par 
tial; if partial, whether by means of a probability sample, 
"cut-off" point, or other plan. Specifications for tha sane 
pling plan for partial coverage Surveys are described in Section 


» 


Ce Frequency and timing. It must be determined whether the survey 
is to be single-time, recurrent, or periodic, and 1f periodic 
whether it should be made at nonthly, quarterly, annual or 
other intervals, The timing of a survey should be considered 
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in relation to what is known about cyclical or other variations 
over time in the data being studied, so that the survey results 
may be of maximum uses , 


d. Method of collection. It mst be determined whether the informa- 
ion is to be collected by mail, personal interview, telephone, 
telegraph, or other method, or by some combination of these 
methods . 


e. Consideration of nonsampling errors, In advance of the final de- 
cision on method, an attempt should be made to identify the main 
sources of bias and of accidental and nonrandom errors antici~ 
pated from alternative procedures, and to indicate the difficul~ 
ties that may arise from inadequate coverage of the respondent 
groupe Plans for testing the reliability of the data and study- 
ing errors of response and other nonsampling errors, including 
interviewer bias, should be incorporated in the basic project 
plan. 


f£. Standard definitions and classifications. Wherever applicable, 
the standard definitions and classifications issued by the 
Bureau of the Budget should be used in the survey plans, so 
that the resulting data may be consistent with data from other 
statistical projects. These standard definitions and classifi- 
cations include: 


Standard Commodity Classification 

Standard Industrial Classification 

Standard Definitions of Metropolitan Areas 

Standard Definitions of Employment and Production Workera 
Standard Payroll-reporting Period 


@- Processing and interpretation of the data. The basic design of 
the survey plan showld includes 

Procedures for editing and coding--methods proposed, codes, 
records, and provisions for evaluation of coding procedures 

Tabulation plans, illustrated by dummy tables and describ-— 
ing method of showing "unknowns" and refusals (e.g., whether 
distributed or shown separately) 

Processing plans-~whether manual or mechanical, and provision 
for controlling the errors of processing 

Outline plans for the analysis and interpretation of results 


h. Allowance for pretests and follow-ups. Development of the sur- 


vey plan should include allowance for pretests needed to test 
the feasibility of the plan, and for follow-ups necessary to 
increase the rate of response and the accuracy of the survey 
results. Recommendations for the planning of pretests are 
presented in Section 5, and for follow-ups in Section 6. 
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i. Comparison with data from other sources. Plans should be made 
for comparison of the data obtained in the survey with data 
from independent sources, Although such comparisons cannot 
be used as validation of the survey results, they do often 
afford information that is extremely useful in the interpre- 
tation of the results. 


j. Proposed calendar. Approximate dates for the following steps 
in the conduct of the survey should be determined in advance 
to ensure orderly completion of the project: 


Beginning of the preliminary work on the design of the survey 
Beginning and completion of the pretest 
Beginning of the field work or mailing of questionnaires 
Completion of the field work or due date for questionnaires 
Completion of the editing and coding 
Completion of the tabulations 

Completion of the preliminary report 

Completion of the final report, including evaluation of the 
variances and biases of the survey results . 


k. Cost estimates. Estimates should be made of the anticipated 
total costs of the survey, including those for: 


Preliminary and exploratory work 

Development and printing of questionnaires, definitions 
and instructions 

Manuals and other instructions for the conduct of the survey 

‘Collection (mailing costs or field costs, including inter- 
viewing, travel, and supervision) 

Editing and coding 

Tabulation 

Analysis and preparation of final report ~ 

Overhead and other costs 


7 


h. Questionnaire and accompanying instructions 


The questionnaire or report form and accompanying instructions should 
be designed 1n accordance with the general principles set forth in "Stand- 
ards for the Design of Report Forms! (Bureau of the Budget, August 19)5-- 
to be reissued as one of the Exhibits to Circular No. A-46). This mammal, 
which applies to administrative as well as to statistical forms, includes 
recommendations relating to specific items on the form and guides on 
layout and printing. 


5 Pretests 


Tt is desirable to test the feasibility of most survey plans in. ad- 
vance, Pretests should be designed and conducted to obtain answers to 
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a limited number of explicit questions on such topics as: 


Relative effectiveness and costs of alternative questionnaires, 
instructions, and operating procedures 

Acceptability and intelligibility of the questions from the respond~ 
ent's point of view 

Possible misunderstandings of questions and procedure on the part 
of the interviewers 

Clarity and applicability of definitions and classifications; 
reference dates 

. Completeness of questions for correct coding and interpretation 

Defects in the forms, maps, lists, instructions, etc. 

Estimates of strata means and variances 

Response rates 


Without proper statistical design the pretest may fail to give answers 
to these questions. 


The pretest should be kept as simple as possible so that clearcut 
answers to the questions being tested will not be obscured by adminis- 
trative complexities, This criterion, with consideration of the specific 
objectives to be served and the cost and time elements involved, should 
determine the choice of classes or localities in which the pretest is to 
be conducted; the approximate size of the pretest (number of interviews); 
the sampling plan within the classes or localities selected; the maps, 
lists or directories to be used; the pretest calendar, including date of 
commencement and date when tabulations are to be made available; the re- 
cords to be kept; the tabulations required; and the analyses to be made. 


The tabulations should be limited to those necessary to answer the 
specific questions for which the pretest was designed, and should gen- 
erally not include additional tabulations for research purposes or for 
preliminary results for which the pretest data may not be valid. 


Adequate time should be allowed for the results of the pretest to 
be analyzed and used to modify the plans for the main survey. * 


6 Follow-ups 


In most surveys response to the first attempt té collect the infor- 
mation, whether by mail or personal enumeration, is insufficient for final 
estimates, and a procedure of follow-up is needed. In planning the num- 
ber and timing of follow-ups, consideration should be given to the esti-~ 
mated rate of response to the initial inquiry and to the precision required. 
The method of collection used in successive follos-ups-~whether by mall, 
eall-back interview, or other means-~should be determined by the response 
vates which can be anticipated from alternative methods, considered in 
relation to the costs involved. 
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Analysis of snecessive follow-ups and information on characteristics 
of those not responding initially is generally needed to aid in preparing 
final estimates from the survey. Information on characteristics obtained 
from respondents unable or unwilling to provide the information called 
for on the questionnaire may be useful in the interpretation of the data. 


7~ Development of the sampling plan for partial coverage surveys 


In partial coverage surveys, the development of the mailing or in~ 

terview list should be carefully defined and fully described, The first 
‘step in developing the list of respondents is to define the type of samn- 
ple design—~i.e., whether a probability sample, a case study of type 
situations, complete enumeration of some previously developed listing, 

a "cut-off" (inclusion of all cases beyond a specified criterion, such 
as size of business, and of no cases below that criterion), or some other 
plane 


Selection of the particular method of partial coverage depends on 
the purposes to be served. Probability samples are necessary where un- 
known biases of selection and estimation will be hazardous and where it 

-is desirable to know the precision of final results. Use of the cut-off 
point often produces the data needed at minimum cost and burden. Surveys 
using cut-off points will be most efficient if designed with the aid of 
a cost-funection and the theory of probability to achleve a minimum mean 
square error. Reasons for selection of a particular method of partial 
coverage should be included in the description of the sampling plan. 


_ Qnce the methed of partial coverage has been defined, a description 
of the sample design should be prepared, the universe defined, along 
with the "framett.or list of units from which the sample may be selected, 
the basis for grouping these units, the technique of selecting partbicu- 
lar units in each group for inclusion in the sample, and the methods for 
summarizing information collected from those units and drawing inferences 
about what has been revealed by the survey. 


The description of the sample design might be checked against the 
outline below, Tha outline is directly applicable to probability samples, 
but many of the items are relevant to other designs as well. The termi- 
nology used in the outline is consistent with that, defined in "The Prep-. 
aration of oe Survey Reports" (United Nations Statistical FADER Es 
Series C, No. 1 (Revised), 2 February 1950). 


a, Sampling units 
The primary units (description and mmber in universe) 
The sampling units to be used at the second and — stages 
(description and number in universe) 
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Lists, maps, directories and other aids that will define and 
delineate the sampling units. Construction of lists in the 
field. 


Criteria of stratification 

The cost function 

The formula for the variance . 

The optimum allocation of the sample 


Method of drawing the sample at each stage, including the ratio 
sampled, by stratum and by stage, and the technique of identi- 
fying the individual sample respondents (e.g., by use of random 
numbers, or by systematic selection from a random start) 


Form of estimate proposed, for each variate of major importance--— 


Formula for the estimate; source of benchmark data if em- 
ployed; waights, derivation 

Universe parameters (proportions, variances, correlations, 
and their effect on the sample design) 

Numerical evaluation of any bias in the formula of esti- 
mation 

Formula for the expected variance 

Numerical approximation to. the variance 


Statement regarding the proposed control and measurement of 
biases—— 


“Methods proposed for reducing nonresponse 

Revisits to measure failures and defects arising trai 
under-coverage and over—coverage; misunderstanding the 
questionnaire and instructions; interviewers! biases; 
incorrect definition and classification; various kinds 
of errors of reporting 


8. Supervision of field enumeration 


In large-scale statistical surveys made by personal enumeration, 
it is particularly important to develop well organized procedures to 
aid in achieving good performance and maximm consistency. In estab- 
lishing procedures for the conduct and eee ere of field work, con~ 
sideration should be given tos 


Be 


, 


‘Methods of selection, training and supervision of the enumer— 


ators, 
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Specifications for training field workers, including mimber of 
hours of training, methods and devices used for training pur-. 
poses, whether payment is made for training sessions. 


Organization for the supervision of the field work, including 
qualifications and experience of the supervisors, method and 
frequency of communication between central and field offices. 


Basis of payment to enumerators (piece-work, hourly, full-time, 
etc.) and scale of wages and salary. 


Plans for measuring differences among enumerators. 


Plans for measuring major types of biases, including those a- 
rising from the enumerators, the questionnaire design, timing, 
and respondents, 


Anticipated rate of response; differential rate by type of 
respondent. 


Specification of what method or methods will be used to deal 
with nonresponse--the number of call-—backs required end the 
procedure for making them; the method of correcting for vary~ 
ing probabilities of individuals being at home; subsampling 
of nonrespondents; or other techniques for dealing with non- 
response. 


Provision for central control in large field studies, and es- 
tablishment of procedures to insure uniform and consistent 
rulings on matters not covered by instructions. 


9, Manuals and other instructions for the conduct of the survey 


Instructions should be developed for all phases of the survey to ex- 
plain the basic definitions and set forth in detail the procedures to be 
followed. Depending on the magnitude and type of survey, materials which 
should be prepared include: ~ 


Specifications for selection and training of enumerators, field 
supervisors, editors, coders, analysts, etd, 

Manual of instructions for field workers, including procedures for 
listing and for selecting the sample, training kits, etc. 

Manuals of instructions for editors, coders, machine operators, etc. 


10. Progress and cost reporting 


In large-scale statistical surveys, there should be systematic 
pians for reporting progress on each phase of the survey, to determine 


(No. A~16) 


Approved For Release 1999/09/25 : CIA-RDP86-00244R000300400016-3 


oe mmmmnmemmmmen! 


- Approved For Release 1999/09/25 :*CIA-RDP86-00244R000300400016-3 


whether the allowances of time and funds are being met and make possible 
any necessary modifications in the survey calendar. These reports should 
also call attention to particular problems encountered, to aid in evalu- 
ating final results and in planning other surveys. Items which should 
be included in the periodic reports are: 


Progress of the work, in the field and in the office : 

Description of any special difficulties or costs encountered in 
the field; proposed remedies 

Description of any special difficulties encountered in the office; 
proposed remedies 

‘ Description of difficulties encountered with the definitions 

Identification of any special biases encountered (nonresponse, 
failure to understand the questions, failure of the interviewers 
to understand the definitions and sampling procedures, etc.) 


At the conclusion of every survey a cost analysis should be made, 
covering the items listed in Section h, paragraph k. The final costs 
should be compared with the advance estimates and summarized for future 
USE. 


il. Preparation and publication of the final report 


Good statistical practice for the conduct of a survey includes, 
finally, a careful presentation of the results. Attention should be 
given in preparing the final report on the survey to the "Standards for 
the Publication of Statistical Data." These standards, issued in 19h7, 
were developed by the Bureau of the Budget in cooperation with the major 

' statistical agencies of the Government to reduce the areas of possible 
misunderstanding or misinterpretation of statistical data. They will 
be reissued as one of the Exhibits to Circular No. A-l6. 
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